{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SageMaker Custom Scorer Evaluation - Demo\n",
    "\n",
    "This notebook demonstrates how to use the CustomScorerEvaluator to evaluate models with custom evaluator functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Import necessary modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.train.evaluate import CustomScorerEvaluator\n",
    "from rich.pretty import pprint\n",
    "\n",
    "# Configure logging to show INFO messages\n",
    "import logging\n",
    "logging.basicConfig(\n",
    "    level=logging.INFO,\n",
    "    format='%(levelname)s - %(name)s - %(message)s'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure Evaluation Parameters\n",
    "\n",
    "Set up the parameters for your custom scorer evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Configuration:\n",
      "  Evaluator: arn:aws:sagemaker:us-west-2:052150106756:hub-content/F3LMYANDKWPZCROJVCKMJ7TOML6QMZBZRRQOVTUL45VUK7PJ4SXA/JsonDoc/eval-lambda-test/0.0.1\n",
      "  Dataset: s3://sagemaker-us-west-2-052150106756/studio-users/d20251107t195443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl\n",
      "  Base Model: arn:aws:sagemaker:us-west-2:052150106756:model-package/test-finetuned-models-gamma/28\n",
      "  Output Location: s3://mufi-test-serverless-smtj/eval/\n"
     ]
    }
   ],
   "source": [
    "# Evaluator ARN (custom evaluator from AI Registry)\n",
    "# evaluator_arn = \"arn:aws:sagemaker:us-west-2:052150106756:hub-content/AIRegistry/JsonDoc/00-goga-qa-evaluation/1.0.0\"\n",
    "# evaluator_arn = \"arn:aws:sagemaker:us-west-2:052150106756:hub-content/AIRegistry/JsonDoc/nikmehta-reward-function/1.0.0\"\n",
    "# evaluator_arn = \"arn:aws:sagemaker:us-west-2:052150106756:hub-content/AIRegistry/JsonDoc/eval-lambda-test/0.0.1\"\n",
    "evaluator_arn = \"arn:aws:sagemaker:us-west-2:052150106756:hub-content/F3LMYANDKWPZCROJVCKMJ7TOML6QMZBZRRQOVTUL45VUK7PJ4SXA/JsonDoc/eval-lambda-test/0.0.1\"\n",
    "\n",
    "# Dataset - can be S3 URI or AIRegistry DataSet ARN\n",
    "dataset = \"s3://sagemaker-us-west-2-052150106756/studio-users/d20251107t195443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl\"\n",
    "\n",
    "# Base model - can be:\n",
    "# 1. Model package ARN: \"arn:aws:sagemaker:region:account:model-package/name/version\"\n",
    "# 2. JumpStart model ID: \"llama-3-2-1b-instruct\" [Evaluation with Base Model Only is yet to be implemented/tested - Not Working currently]\n",
    "base_model = \"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-finetuned-models-gamma/28\"\n",
    "\n",
    "# S3 location for outputs\n",
    "s3_output_path = \"s3://mufi-test-serverless-smtj/eval/\"\n",
    "\n",
    "# Optional: MLflow tracking server ARN\n",
    "mlflow_resource_arn = \"arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server/mmlu-eval-experiment\"\n",
    "\n",
    "print(\"Configuration:\")\n",
    "print(f\"  Evaluator: {evaluator_arn}\")\n",
    "print(f\"  Dataset: {dataset}\")\n",
    "print(f\"  Base Model: {base_model}\")\n",
    "print(f\"  Output Location: {s3_output_path}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create CustomScorerEvaluator Instance\n",
    "\n",
    "Instantiate the evaluator with your configuration. The evaluator can accept:\n",
    "- **Custom Evaluator ARN** (string): Points to your custom evaluator in AI Registry\n",
    "- **Built-in Metric** (string or enum): Use preset metrics like \"code_executions\", \"math_answers\", etc.\n",
    "- **Evaluator Object**: A sagemaker.ai_registry.evaluator.Evaluator instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 13:42:33] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Found credentials in shared credentials file: ~<span style=\"color: #e100e1; text-decoration-color: #e100e1\">/.aws/credentials</span>   <a href=\"file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/botocore/credentials.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">credentials.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/botocore/credentials.py#1364\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">1364</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 13:42:33]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Found credentials in shared credentials file: ~\u001b[38;2;225;0;225m/.aws/\u001b[0m\u001b[38;2;225;0;225mcredentials\u001b[0m   \u001b]8;id=639873;file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/botocore/credentials.py\u001b\\\u001b[2mcredentials.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=963387;file:///Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/botocore/credentials.py#1364\u001b\\\u001b[2m1364\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sagemaker.config INFO - Not applying SDK defaults from location: /Library/Application Support/sagemaker/config.yaml\n",
      "sagemaker.config INFO - Not applying SDK defaults from location: /Users/mufi/Library/Application Support/sagemaker/config.yaml\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Resolved MLflow resource ARN:                                    <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#113\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">113</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server/ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         mmlu-eval-experiment                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Resolved MLflow resource ARN:                                    \u001b]8;id=342593;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=318918;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#113\u001b\\\u001b[2m113\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server/ \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         mmlu-eval-experiment                                             \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "✓ CustomScorerEvaluator created successfully\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">CustomScorerEvaluator</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">region</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">sagemaker_session</span>=<span style=\"font-weight: bold\">&lt;</span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">sagemaker.core.helper.session_helper.Session</span><span style=\"color: #000000; text-decoration-color: #000000\"> object at </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0x116ae9f40</span><span style=\"font-weight: bold\">&gt;</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">model</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:model-package/test-finetuned-models-gamma/28'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">base_eval_name</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'eval-meta-1b49b716'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">s3_output_path</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'s3://mufi-test-serverless-smtj/eval/'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">mlflow_resource_arn</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server/mmlu-eval-experiment'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">mlflow_experiment_name</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">mlflow_run_name</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">networking</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">kms_key_id</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">model_package_group</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">evaluator</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:hub-content/F3LMYANDKWPZCROJVCKMJ7TOML6QMZBZRRQOVTUL45VUK7PJ4SXA/JsonDoc/eval-lambda-test/0.0.1'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">dataset</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'s3://sagemaker-us-west-2-052150106756/studio-users/d20251107t195443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">evaluate_base_model</span>=<span style=\"color: #d70000; text-decoration-color: #d70000; font-style: italic\">False</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;38;2;225;0;225mCustomScorerEvaluator\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mregion\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0msagemaker_session\u001b[0m=\u001b[1m<\u001b[0m\u001b[1;38;2;225;0;225msagemaker.core.helper.session_helper.Session\u001b[0m\u001b[39m object at \u001b[0m\u001b[1;36m0x116ae9f40\u001b[0m\u001b[1m>\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmodel\u001b[0m=\u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:model-package/test-finetuned-models-gamma/28'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mbase_eval_name\u001b[0m=\u001b[38;2;0;135;0m'eval-meta-1b49b716'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0ms3_output_path\u001b[0m=\u001b[38;2;0;135;0m's3://mufi-test-serverless-smtj/eval/'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmlflow_resource_arn\u001b[0m=\u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server/mmlu-eval-experiment'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmlflow_experiment_name\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmlflow_run_name\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mnetworking\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mkms_key_id\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mmodel_package_group\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mevaluator\u001b[0m=\u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:hub-content/F3LMYANDKWPZCROJVCKMJ7TOML6QMZBZRRQOVTUL45VUK7PJ4SXA/JsonDoc/eval-lambda-test/0.0.1'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mdataset\u001b[0m=\u001b[38;2;0;135;0m's3://sagemaker-us-west-2-052150106756/studio-users/d20251107t195443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mevaluate_base_model\u001b[0m=\u001b[3;38;2;215;0;0mFalse\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create evaluator with custom evaluator ARN\n",
    "evaluator = CustomScorerEvaluator(\n",
    "    evaluator=evaluator_arn,  # Custom evaluator ARN\n",
    "    dataset=dataset,\n",
    "    model=base_model,\n",
    "    s3_output_path=s3_output_path,\n",
    "    mlflow_resource_arn=mlflow_resource_arn,\n",
    "    # model_package_group=\"arn:aws:sagemaker:us-west-2:052150106756:model-package-group/Demo-test-deb-2\", \n",
    "    evaluate_base_model=False  # Set to True to also evaluate the base model\n",
    ")\n",
    "\n",
    "print(\"\\n✓ CustomScorerEvaluator created successfully\")\n",
    "pprint(evaluator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optionally update the hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 13:42:38] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> SageMaker Python SDK will collect telemetry to help us better  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">telemetry_logging.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py#91\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">91</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         understand our user's needs, diagnose issues, and deliver      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         additional features.                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         To opt out of telemetry, please disable via TelemetryOptOut    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         parameter in SDK defaults config. For more information, refer  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         to                                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #0069ff; text-decoration-color: #0069ff; text-decoration: underline\">https://sagemaker.readthedocs.io/en/stable/overview.html#confi</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #0069ff; text-decoration-color: #0069ff; text-decoration: underline\">guring-and-using-defaults-with-the-sagemaker-python-sdk.</span>       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 13:42:38]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m SageMaker Python SDK will collect telemetry to help us better  \u001b]8;id=848286;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py\u001b\\\u001b[2mtelemetry_logging.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=998219;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py#91\u001b\\\u001b[2m91\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         understand our user's needs, diagnose issues, and deliver      \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         additional features.                                           \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         To opt out of telemetry, please disable via TelemetryOptOut    \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         parameter in SDK defaults config. For more information, refer  \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         to                                                             \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255mhttps://sagemaker.readthedocs.io/en/stable/overview.html#confi\u001b[0m \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255mguring-and-using-defaults-with-the-sagemaker-python-sdk.\u001b[0m       \u001b[2m                       \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Fetching evaluation override parameters for             <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/custom_scorer_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">custom_scorer_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/custom_scorer_evaluator.py#236\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">236</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         hyperparameters property                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                              </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Fetching evaluation override parameters for             \u001b]8;id=20210;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/custom_scorer_evaluator.py\u001b\\\u001b[2mcustom_scorer_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=113368;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/custom_scorer_evaluator.py#236\u001b\\\u001b[2m236\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         hyperparameters property                                \u001b[2m                              \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Fetching hub content metadata for                                  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">recipe_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#201\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">201</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         meta-textgeneration-llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>-1b-instruct from SageMakerPublicHub  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                   </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Fetching hub content metadata for                                  \u001b]8;id=402391;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\u001b\\\u001b[2mrecipe_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=385188;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#201\u001b\\\u001b[2m201\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         meta-textgeneration-llama-\u001b[1;36m3\u001b[0m-\u001b[1;36m2\u001b[0m-1b-instruct from SageMakerPublicHub  \u001b[2m                   \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #d7af00; text-decoration-color: #d7af00; font-weight: bold\">WARNING </span> No region provided. Using default region.                                 <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/utils/utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/utils/utils.py#340\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">340</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;215;175;0mWARNING \u001b[0m No region provided. Using default region.                                 \u001b]8;id=442028;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/utils/utils.py\u001b\\\u001b[2mutils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=947914;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/utils/utils.py#340\u001b\\\u001b[2m340\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Runs on sagemaker us-west-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>, region:us-west-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>                             <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/utils/utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/utils/utils.py#354\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">354</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Runs on sagemaker us-west-\u001b[1;36m2\u001b[0m, region:us-west-\u001b[1;36m2\u001b[0m                             \u001b]8;id=708289;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/utils/utils.py\u001b\\\u001b[2mutils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=968385;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/utils/utils.py#354\u001b\\\u001b[2m354\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Searching for evaluation recipe with <span style=\"color: #d7af00; text-decoration-color: #d7af00\">Type</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Evaluation'</span> and         <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">recipe_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#221\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">221</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #d7af00; text-decoration-color: #d7af00\">EvaluationType</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'DeterministicEvaluation'</span>                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                   </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Searching for evaluation recipe with \u001b[38;2;215;175;0mType\u001b[0m=\u001b[38;2;0;135;0m'Evaluation'\u001b[0m and         \u001b]8;id=711157;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\u001b\\\u001b[2mrecipe_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=750371;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#221\u001b\\\u001b[2m221\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;215;175;0mEvaluationType\u001b[0m=\u001b[38;2;0;135;0m'DeterministicEvaluation'\u001b[0m                           \u001b[2m                   \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Downloading override parameters from                               <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">recipe_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#249\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">249</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         s3:<span style=\"color: #e100e1; text-decoration-color: #e100e1\">//jumpstart-cache-beta-us-west-2/recipes/open-source-eval-meta-</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                   </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #e100e1; text-decoration-color: #e100e1\">textgeneration-llama-3-2-1b-instruct-deterministic_override_params</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                   </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #e100e1; text-decoration-color: #e100e1\">_sm_jobs_v1.0.19.json</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                   </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Downloading override parameters from                               \u001b]8;id=762518;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py\u001b\\\u001b[2mrecipe_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=755839;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/recipe_utils.py#249\u001b\\\u001b[2m249\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         s3:\u001b[38;2;225;0;225m/\u001b[0m\u001b[38;2;225;0;225m/jumpstart-cache-beta-us-west-2/recipes/\u001b[0m\u001b[38;2;225;0;225mopen-source-eval-meta-\u001b[0m \u001b[2m                   \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;225;0;225mtextgeneration-llama-3-2-1b-instruct-deterministic_override_params\u001b[0m \u001b[2m                   \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;225;0;225m_sm_jobs_v1.0.19.json\u001b[0m                                              \u001b[2m                   \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'max_new_tokens'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'8192'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'temperature'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'0'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'top_k'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'-1'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'top_p'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'1.0'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'aggregation'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">''</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'postprocessing'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'False'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #008700; text-decoration-color: #008700\">'max_model_len'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'12000'</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'max_new_tokens'\u001b[0m: \u001b[38;2;0;135;0m'8192'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'temperature'\u001b[0m: \u001b[38;2;0;135;0m'0'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'top_k'\u001b[0m: \u001b[38;2;0;135;0m'-1'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'top_p'\u001b[0m: \u001b[38;2;0;135;0m'1.0'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'aggregation'\u001b[0m: \u001b[38;2;0;135;0m''\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'postprocessing'\u001b[0m: \u001b[38;2;0;135;0m'False'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;0;135;0m'max_model_len'\u001b[0m: \u001b[38;2;0;135;0m'12000'\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pprint(evaluator.hyperparameters.to_dict())\n",
    "\n",
    "# optionally update hyperparameters\n",
    "# evaluator.hyperparameters.temperature = \"0.1\"\n",
    "\n",
    "# optionally get more info on types, limits, defaults.\n",
    "# evaluator.hyperparameters.get_info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Alternative: Using Built-in Metrics\n",
    "\n",
    "Instead of a custom evaluator ARN, you can use built-in metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example with built-in metrics (commented out)\n",
    "# from sagemaker.train.evaluate import get_builtin_metrics\n",
    "# \n",
    "# BuiltInMetric = get_builtin_metrics()\n",
    "# \n",
    "# evaluator_builtin = CustomScorerEvaluator(\n",
    "#     evaluator=BuiltInMetric.PRIME_MATH,  # Or use string: \"prime_math\"\n",
    "#     dataset=dataset,\n",
    "#     base_model=base_model,\n",
    "#     s3_output_path=s3_output_path\n",
    "# )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start Evaluation\n",
    "\n",
    "Call `evaluate()` to start the evaluation job. This will:\n",
    "1. Create or update the evaluation pipeline\n",
    "2. Start a pipeline execution\n",
    "3. Return an `EvaluationPipelineExecution` object for monitoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 13:42:43] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> SageMaker Python SDK will collect telemetry to help us better  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">telemetry_logging.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py#91\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">91</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         understand our user's needs, diagnose issues, and deliver      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         additional features.                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         To opt out of telemetry, please disable via TelemetryOptOut    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         parameter in SDK defaults config. For more information, refer  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         to                                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #0069ff; text-decoration-color: #0069ff; text-decoration: underline\">https://sagemaker.readthedocs.io/en/stable/overview.html#confi</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #0069ff; text-decoration-color: #0069ff; text-decoration: underline\">guring-and-using-defaults-with-the-sagemaker-python-sdk.</span>       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 13:42:43]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m SageMaker Python SDK will collect telemetry to help us better  \u001b]8;id=201476;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py\u001b\\\u001b[2mtelemetry_logging.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=125527;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py#91\u001b\\\u001b[2m91\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         understand our user's needs, diagnose issues, and deliver      \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         additional features.                                           \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         To opt out of telemetry, please disable via TelemetryOptOut    \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         parameter in SDK defaults config. For more information, refer  \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         to                                                             \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255mhttps://sagemaker.readthedocs.io/en/stable/overview.html#confi\u001b[0m \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255mguring-and-using-defaults-with-the-sagemaker-python-sdk.\u001b[0m       \u001b[2m                       \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Getting or creating artifact for source:                         <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#597\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">597</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fine <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         tuned-models-gamma/<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">28</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Getting or creating artifact for source:                         \u001b]8;id=336129;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=429516;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#597\u001b\\\u001b[2m597\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fine \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         tuned-models-gamma/\u001b[1;36m28\u001b[0m                                            \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Searching for existing artifact for model package:               <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#459\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">459</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fine <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         tuned-models-gamma/<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">28</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Searching for existing artifact for model package:               \u001b]8;id=916341;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=92767;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#459\u001b\\\u001b[2m459\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fine \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         tuned-models-gamma/\u001b[1;36m28\u001b[0m                                            \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Found existing artifact:                                         <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#468\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">468</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b3 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         138877d772ec489bef                                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Found existing artifact:                                         \u001b]8;id=110957;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=865654;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#468\u001b\\\u001b[2m468\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b3 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         138877d772ec489bef                                               \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Inferred model package group ARN:                                <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#386\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">386</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package-group/tes <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         t-finetuned-models-gamma from                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fine <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         tuned-models-gamma/<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">28</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Inferred model package group ARN:                                \u001b]8;id=126121;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=198580;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#386\u001b\\\u001b[2m386\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package-group/tes \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         t-finetuned-models-gamma from                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fine \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         tuned-models-gamma/\u001b[1;36m28\u001b[0m                                            \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Automatically inferred model_package_group:                      <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#421\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">421</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package-group/tes <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         t-finetuned-models-gamma                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Automatically inferred model_package_group:                      \u001b]8;id=183930;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=417470;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#421\u001b\\\u001b[2m421\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package-group/tes \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         t-finetuned-models-gamma                                         \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Using ModelPackage - model_package_group_arn:           <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/custom_scorer_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">custom_scorer_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/custom_scorer_evaluator.py#421\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">421</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package- <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                              </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         group/test-finetuned-models-gamma                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                              </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Using ModelPackage - model_package_group_arn:           \u001b]8;id=191140;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/custom_scorer_evaluator.py\u001b\\\u001b[2mcustom_scorer_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=51752;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/custom_scorer_evaluator.py#421\u001b\\\u001b[2m421\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package- \u001b[2m                              \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         group/test-finetuned-models-gamma                       \u001b[2m                              \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Resolved model info - base_model_name:                  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/custom_scorer_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">custom_scorer_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/custom_scorer_evaluator.py#424\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">424</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         meta-textgeneration-llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>-1b-instruct,              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                              </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         base_model_arn:                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                              </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPu <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                              </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         blicHub/Model/meta-textgeneration-llama-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>-1b-instruct <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                              </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         /<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.10</span>.<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>, source_model_package_arn:                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                              </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:model-package/ <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                              </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         test-finetuned-models-gamma/<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">28</span>                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                              </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Resolved model info - base_model_name:                  \u001b]8;id=359160;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/custom_scorer_evaluator.py\u001b\\\u001b[2mcustom_scorer_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=935533;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/custom_scorer_evaluator.py#424\u001b\\\u001b[2m424\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         meta-textgeneration-llama-\u001b[1;36m3\u001b[0m-\u001b[1;36m2\u001b[0m-1b-instruct,              \u001b[2m                              \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         base_model_arn:                                         \u001b[2m                              \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPu \u001b[2m                              \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         blicHub/Model/meta-textgeneration-llama-\u001b[1;36m3\u001b[0m-\u001b[1;36m2\u001b[0m-1b-instruct \u001b[2m                              \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         /\u001b[1;36m1.10\u001b[0m.\u001b[1;36m0\u001b[0m, source_model_package_arn:                      \u001b[2m                              \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:model-package/ \u001b[2m                              \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         test-finetuned-models-gamma/\u001b[1;36m28\u001b[0m                          \u001b[2m                              \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> SageMaker Python SDK will collect telemetry to help us better  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">telemetry_logging.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py#91\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">91</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         understand our user's needs, diagnose issues, and deliver      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         additional features.                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         To opt out of telemetry, please disable via TelemetryOptOut    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         parameter in SDK defaults config. For more information, refer  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         to                                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #0069ff; text-decoration-color: #0069ff; text-decoration: underline\">https://sagemaker.readthedocs.io/en/stable/overview.html#confi</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #0069ff; text-decoration-color: #0069ff; text-decoration: underline\">guring-and-using-defaults-with-the-sagemaker-python-sdk.</span>       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                       </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m SageMaker Python SDK will collect telemetry to help us better  \u001b]8;id=189431;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py\u001b\\\u001b[2mtelemetry_logging.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=22751;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/telemetry/telemetry_logging.py#91\u001b\\\u001b[2m91\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         understand our user's needs, diagnose issues, and deliver      \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         additional features.                                           \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         To opt out of telemetry, please disable via TelemetryOptOut    \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         parameter in SDK defaults config. For more information, refer  \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         to                                                             \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255mhttps://sagemaker.readthedocs.io/en/stable/overview.html#confi\u001b[0m \u001b[2m                       \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[4;38;2;0;105;255mguring-and-using-defaults-with-the-sagemaker-python-sdk.\u001b[0m       \u001b[2m                       \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Using configured hyperparameters: <span style=\"font-weight: bold\">{</span><span style=\"color: #008700; text-decoration-color: #008700\">'max_new_tokens'</span>:    <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/custom_scorer_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">custom_scorer_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/custom_scorer_evaluator.py#299\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">299</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'8192'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'temperature'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'0'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'top_k'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'-1'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'top_p'</span>:     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                              </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'1.0'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'aggregation'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">''</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'postprocessing'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'False'</span>,    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                              </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'max_model_len'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'12000'</span><span style=\"font-weight: bold\">}</span>                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                              </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Using configured hyperparameters: \u001b[1m{\u001b[0m\u001b[38;2;0;135;0m'max_new_tokens'\u001b[0m:    \u001b]8;id=536279;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/custom_scorer_evaluator.py\u001b\\\u001b[2mcustom_scorer_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=194605;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/custom_scorer_evaluator.py#299\u001b\\\u001b[2m299\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'8192'\u001b[0m, \u001b[38;2;0;135;0m'temperature'\u001b[0m: \u001b[38;2;0;135;0m'0'\u001b[0m, \u001b[38;2;0;135;0m'top_k'\u001b[0m: \u001b[38;2;0;135;0m'-1'\u001b[0m, \u001b[38;2;0;135;0m'top_p'\u001b[0m:     \u001b[2m                              \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'1.0'\u001b[0m, \u001b[38;2;0;135;0m'aggregation'\u001b[0m: \u001b[38;2;0;135;0m''\u001b[0m, \u001b[38;2;0;135;0m'postprocessing'\u001b[0m: \u001b[38;2;0;135;0m'False'\u001b[0m,    \u001b[2m                              \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'max_model_len'\u001b[0m: \u001b[38;2;0;135;0m'12000'\u001b[0m\u001b[1m}\u001b[0m                               \u001b[2m                              \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Using full template for ModelPackage                             <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#655\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">655</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Using full template for ModelPackage                             \u001b]8;id=164880;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=880373;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#655\u001b\\\u001b[2m655\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 13:42:44] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Resolved template parameters: <span style=\"font-weight: bold\">{</span><span style=\"color: #008700; text-decoration-color: #008700\">'role_arn'</span>:                       <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#693\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">693</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:iam::052150106756:role/Admin'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'mlflow_resource_arn'</span>:   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">/mmlu-eval-experiment'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'mlflow_experiment_name'</span>: <span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'mlflow_run_name'</span>: <span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'model_package_group_arn'</span>:              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:model-package-group/te</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">st-finetuned-models-gamma'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'source_model_package_arn'</span>:          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">etuned-models-gamma/28'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'base_model_arn'</span>:                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">Model/meta-textgeneration-llama-3-2-1b-instruct/1.10.0'</span>,         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'s3_output_path'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'s3://mufi-test-serverless-smtj/eval/'</span>,        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'dataset_artifact_arn'</span>:                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">3138877d772ec489bef'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'action_arn_prefix'</span>:                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:action'</span>,               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'dataset_uri'</span>:                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'s3://sagemaker-us-west-2-052150106756/studio-users/d20251107t19</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">5443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'task'</span>:   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'gen_qa'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'strategy'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'gen_qa'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'evaluation_metric'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'all'</span>,      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'pipeline_name'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'SagemakerEvaluation-Deterministic'</span>,            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'evaluate_base_model'</span>: <span style=\"color: #d70000; text-decoration-color: #d70000; font-style: italic\">False</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'evaluator_arn'</span>:                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'arn:aws:sagemaker:us-west-2:052150106756:hub-content/F3LMYANDKW</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">PZCROJVCKMJ7TOML6QMZBZRRQOVTUL45VUK7PJ4SXA/JsonDoc/eval-lambda-t</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">est/0.0.1'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'max_new_tokens'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'8192'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'temperature'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'0'</span>,        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'top_k'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'-1'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'top_p'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'1.0'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'aggregation'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'mean'</span>,            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">'postprocessing'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'True'</span>, <span style=\"color: #008700; text-decoration-color: #008700\">'max_model_len'</span>: <span style=\"color: #008700; text-decoration-color: #008700\">'12000'</span><span style=\"font-weight: bold\">}</span>              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 13:42:44]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Resolved template parameters: \u001b[1m{\u001b[0m\u001b[38;2;0;135;0m'role_arn'\u001b[0m:                       \u001b]8;id=863350;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=151185;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#693\u001b\\\u001b[2m693\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:iam::052150106756:role/Admin'\u001b[0m, \u001b[38;2;0;135;0m'mlflow_resource_arn'\u001b[0m:   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m/mmlu-eval-experiment'\u001b[0m, \u001b[38;2;0;135;0m'mlflow_experiment_name'\u001b[0m: \u001b[3;38;2;225;0;225mNone\u001b[0m,          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'mlflow_run_name'\u001b[0m: \u001b[3;38;2;225;0;225mNone\u001b[0m, \u001b[38;2;0;135;0m'model_package_group_arn'\u001b[0m:              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:model-package-group/te\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mst-finetuned-models-gamma'\u001b[0m, \u001b[38;2;0;135;0m'source_model_package_arn'\u001b[0m:          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0metuned-models-gamma/28'\u001b[0m, \u001b[38;2;0;135;0m'base_model_arn'\u001b[0m:                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mModel/meta-textgeneration-llama-3-2-1b-instruct/1.10.0'\u001b[0m,         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m's3_output_path'\u001b[0m: \u001b[38;2;0;135;0m's3://mufi-test-serverless-smtj/eval/'\u001b[0m,        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'dataset_artifact_arn'\u001b[0m:                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m3138877d772ec489bef'\u001b[0m, \u001b[38;2;0;135;0m'action_arn_prefix'\u001b[0m:                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:action'\u001b[0m,               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'dataset_uri'\u001b[0m:                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m's3://sagemaker-us-west-2-052150106756/studio-users/d20251107t19\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m5443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl'\u001b[0m, \u001b[38;2;0;135;0m'task'\u001b[0m:   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'gen_qa'\u001b[0m, \u001b[38;2;0;135;0m'strategy'\u001b[0m: \u001b[38;2;0;135;0m'gen_qa'\u001b[0m, \u001b[38;2;0;135;0m'evaluation_metric'\u001b[0m: \u001b[38;2;0;135;0m'all'\u001b[0m,      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'pipeline_name'\u001b[0m: \u001b[38;2;0;135;0m'SagemakerEvaluation-Deterministic'\u001b[0m,            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'evaluate_base_model'\u001b[0m: \u001b[3;38;2;215;0;0mFalse\u001b[0m, \u001b[38;2;0;135;0m'evaluator_arn'\u001b[0m:                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'arn:aws:sagemaker:us-west-2:052150106756:hub-content/F3LMYANDKW\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mPZCROJVCKMJ7TOML6QMZBZRRQOVTUL45VUK7PJ4SXA/JsonDoc/eval-lambda-t\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mest/0.0.1'\u001b[0m, \u001b[38;2;0;135;0m'max_new_tokens'\u001b[0m: \u001b[38;2;0;135;0m'8192'\u001b[0m, \u001b[38;2;0;135;0m'temperature'\u001b[0m: \u001b[38;2;0;135;0m'0'\u001b[0m,        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'top_k'\u001b[0m: \u001b[38;2;0;135;0m'-1'\u001b[0m, \u001b[38;2;0;135;0m'top_p'\u001b[0m: \u001b[38;2;0;135;0m'1.0'\u001b[0m, \u001b[38;2;0;135;0m'aggregation'\u001b[0m: \u001b[38;2;0;135;0m'mean'\u001b[0m,            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m'postprocessing'\u001b[0m: \u001b[38;2;0;135;0m'True'\u001b[0m, \u001b[38;2;0;135;0m'max_model_len'\u001b[0m: \u001b[38;2;0;135;0m'12000'\u001b[0m\u001b[1m}\u001b[0m              \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Rendered pipeline definition:                                    <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">base_evaluator.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#702\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">702</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"font-weight: bold\">{</span>                                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"color: #008700; text-decoration-color: #008700\">\"Version\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"2020-12-01\"</span>,                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"color: #008700; text-decoration-color: #008700\">\"Metadata\"</span>: <span style=\"font-weight: bold\">{}</span>,                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"color: #008700; text-decoration-color: #008700\">\"MlflowConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"color: #008700; text-decoration-color: #008700\">\"MlflowResourceArn\"</span>:                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">/mmlu-eval-experiment\"</span>                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"font-weight: bold\">}</span>,                                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"color: #008700; text-decoration-color: #008700\">\"Parameters\"</span>: <span style=\"font-weight: bold\">[]</span>,                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"color: #008700; text-decoration-color: #008700\">\"Steps\"</span>: <span style=\"font-weight: bold\">[</span>                                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">{</span>                                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"CreateEvaluationAction\"</span>,                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Lineage\"</span>,                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Arguments\"</span>: <span style=\"font-weight: bold\">{</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"Actions\"</span>: <span style=\"font-weight: bold\">[</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ActionName\"</span>: <span style=\"font-weight: bold\">{</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ActionType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Evaluation\"</span>,                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceUri\"</span>:                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">etuned-models-gamma/28\"</span>,                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"ModelPackage\"</span>                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Properties\"</span>: <span style=\"font-weight: bold\">{</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"PipelineExecutionArn\"</span>: <span style=\"font-weight: bold\">{</span>                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionArn\"</span>          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>,                                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"PipelineName\"</span>:                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"SagemakerEvaluation-Deterministic\"</span>                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"Contexts\"</span>: <span style=\"font-weight: bold\">[</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ContextName\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ContextType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"PipelineExecution\"</span>,                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceUri\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionArn\"</span>          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"Associations\"</span>: <span style=\"font-weight: bold\">[</span>                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>,                                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Action\"</span>                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Destination\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>,                                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Context\"</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"AssociationType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"ContributedTo\"</span>                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>,                                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Arn\"</span>:                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">3138877d772ec489bef\"</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Destination\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Arn\"</span>: <span style=\"font-weight: bold\">{</span>                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"/\"</span>,                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:ac</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">tion\"</span>,                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">{</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                               <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">}</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">]</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">}</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"AssociationType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"ContributedTo\"</span>                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>                                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"font-weight: bold\">}</span>                                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">}</span>,                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">{</span>                                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"EvaluateCustomModel\"</span>,                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Training\"</span>,                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Arguments\"</span>: <span style=\"font-weight: bold\">{</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"RoleArn\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:iam::052150106756:role/Admin\"</span>,       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"ModelPackageConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"ModelPackageGroupArn\"</span>:                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:model-package-group/te</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">st-finetuned-models-gamma\"</span>,                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceModelPackageArn\"</span>:                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">etuned-models-gamma/28\"</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"ServerlessJobConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"BaseModelArn\"</span>:                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">Model/meta-textgeneration-llama-3-2-1b-instruct/1.10.0\"</span>,         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"AcceptEula\"</span>: true,                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"JobType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Evaluation\"</span>,                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"EvaluationType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"CustomScorerEvaluation\"</span>,            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"EvaluatorArn\"</span>:                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:hub-content/F3LMYANDKW</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">PZCROJVCKMJ7TOML6QMZBZRRQOVTUL45VUK7PJ4SXA/JsonDoc/eval-lambda-t</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">est/0.0.1\"</span>                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"StoppingCondition\"</span>: <span style=\"font-weight: bold\">{</span>                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"MaxRuntimeInSeconds\"</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">86400</span>                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"HyperParameters\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"task\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"gen_qa\"</span>,                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"strategy\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"gen_qa\"</span>,                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"evaluation_metric\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"all\"</span>,                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"max_new_tokens\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"8192\"</span>,                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"temperature\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"0\"</span>,                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"top_k\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"-1\"</span>,                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"top_p\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"1.0\"</span>,                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"max_model_len\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"12000\"</span>,                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"aggregation\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"mean\"</span>,                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"postprocessing\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"True\"</span>                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"OutputDataConfig\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"S3OutputPath\"</span>:                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"s3://mufi-test-serverless-smtj/eval/\"</span>,                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"color: #008700; text-decoration-color: #008700\">\"CompressionType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"NONE\"</span>                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">}</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"InputDataConfig\"</span>: <span style=\"font-weight: bold\">[</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ChannelName\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"train\"</span>,                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"DataSource\"</span>: <span style=\"font-weight: bold\">{</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"S3DataSource\"</span>: <span style=\"font-weight: bold\">{</span>                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"S3DataType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"S3Prefix\"</span>,                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"S3Uri\"</span>:                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"s3://sagemaker-us-west-2-052150106756/studio-users/d20251107t19</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">5443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl\"</span>            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>                                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"font-weight: bold\">}</span>                                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">}</span>,                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">{</span>                                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"AssociateLineage\"</span>,                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Lineage\"</span>,                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"DependsOn\"</span>: <span style=\"font-weight: bold\">[</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"CreateEvaluationAction\"</span>                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"font-weight: bold\">]</span>,                                                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"color: #008700; text-decoration-color: #008700\">\"Arguments\"</span>: <span style=\"font-weight: bold\">{</span>                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"Artifacts\"</span>: <span style=\"font-weight: bold\">[</span>                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ArtifactName\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"-\"</span>,                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">{</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">}</span>,                                             <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"custom-eval-report\"</span>                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">]</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"ArtifactType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"EvaluationReport\"</span>,                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"SourceUri\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>:                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">\"Steps.EvaluateCustomModel.OutputDataConfig.S3OutputPath\"</span>        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>                                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>,                                                       <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"color: #008700; text-decoration-color: #008700\">\"Associations\"</span>: <span style=\"font-weight: bold\">[</span>                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">{</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Source\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Name\"</span>: <span style=\"font-weight: bold\">{</span>                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"-\"</span>,                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">{</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                               <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">}</span>,                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"custom-eval-report\"</span>                         <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">]</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">}</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>,                                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Type\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Artifact\"</span>                                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"Destination\"</span>: <span style=\"font-weight: bold\">{</span>                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"color: #008700; text-decoration-color: #008700\">\"Arn\"</span>: <span style=\"font-weight: bold\">{</span>                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"color: #008700; text-decoration-color: #008700\">\"Std:Join\"</span>: <span style=\"font-weight: bold\">{</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"On\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"/\"</span>,                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"color: #008700; text-decoration-color: #008700\">\"Values\"</span>: <span style=\"font-weight: bold\">[</span>                                    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"color: #008700; text-decoration-color: #008700\">\"arn:aws:sagemaker:us-west-2:052150106756:ac</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008700; text-decoration-color: #008700\">tion\"</span>,                                                           <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">{</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                               <span style=\"color: #008700; text-decoration-color: #008700\">\"Get\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"Execution.PipelineExecutionId\"</span>     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                             <span style=\"font-weight: bold\">}</span>                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                           <span style=\"font-weight: bold\">]</span>                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                         <span style=\"font-weight: bold\">}</span>                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                       <span style=\"font-weight: bold\">}</span>                                                  <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"font-weight: bold\">}</span>,                                                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                     <span style=\"color: #008700; text-decoration-color: #008700\">\"AssociationType\"</span>: <span style=\"color: #008700; text-decoration-color: #008700\">\"ContributedTo\"</span>                   <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                   <span style=\"font-weight: bold\">}</span>                                                      <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>                 <span style=\"font-weight: bold\">]</span>                                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>               <span style=\"font-weight: bold\">}</span>                                                          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>             <span style=\"font-weight: bold\">}</span>                                                            <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>           <span style=\"font-weight: bold\">]</span>                                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"font-weight: bold\">}</span>                                                                <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                     </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Rendered pipeline definition:                                    \u001b]8;id=395506;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py\u001b\\\u001b[2mbase_evaluator.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=123517;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/base_evaluator.py#702\u001b\\\u001b[2m702\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[1m{\u001b[0m                                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[38;2;0;135;0m\"Version\"\u001b[0m: \u001b[38;2;0;135;0m\"2020-12-01\"\u001b[0m,                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[38;2;0;135;0m\"Metadata\"\u001b[0m: \u001b[1m{\u001b[0m\u001b[1m}\u001b[0m,                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[38;2;0;135;0m\"MlflowConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[38;2;0;135;0m\"MlflowResourceArn\"\u001b[0m:                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:mlflow-tracking-server\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m/mmlu-eval-experiment\"\u001b[0m                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[1m}\u001b[0m,                                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[38;2;0;135;0m\"Parameters\"\u001b[0m: \u001b[1m[\u001b[0m\u001b[1m]\u001b[0m,                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[38;2;0;135;0m\"Steps\"\u001b[0m: \u001b[1m[\u001b[0m                                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m{\u001b[0m                                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[38;2;0;135;0m\"CreateEvaluationAction\"\u001b[0m,                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Lineage\"\u001b[0m,                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Arguments\"\u001b[0m: \u001b[1m{\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"Actions\"\u001b[0m: \u001b[1m[\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ActionName\"\u001b[0m: \u001b[1m{\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ActionType\"\u001b[0m: \u001b[38;2;0;135;0m\"Evaluation\"\u001b[0m,                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"SourceUri\"\u001b[0m:                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0metuned-models-gamma/28\"\u001b[0m,                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"SourceType\"\u001b[0m: \u001b[38;2;0;135;0m\"ModelPackage\"\u001b[0m                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Properties\"\u001b[0m: \u001b[1m{\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"PipelineExecutionArn\"\u001b[0m: \u001b[1m{\u001b[0m                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionArn\"\u001b[0m          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m,                                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"PipelineName\"\u001b[0m:                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"SagemakerEvaluation-Deterministic\"\u001b[0m                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"Contexts\"\u001b[0m: \u001b[1m[\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ContextName\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ContextType\"\u001b[0m: \u001b[38;2;0;135;0m\"PipelineExecution\"\u001b[0m,                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"SourceUri\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionArn\"\u001b[0m          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"Associations\"\u001b[0m: \u001b[1m[\u001b[0m                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m,                                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Action\"\u001b[0m                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Destination\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m,                                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Context\"\u001b[0m                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"AssociationType\"\u001b[0m: \u001b[38;2;0;135;0m\"ContributedTo\"\u001b[0m                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m,                                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Arn\"\u001b[0m:                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:artifact/2b64ef9fe915b\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m3138877d772ec489bef\"\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Destination\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Arn\"\u001b[0m: \u001b[1m{\u001b[0m                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"/\"\u001b[0m,                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:ac\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mtion\"\u001b[0m,                                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m{\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                               \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m}\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m]\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m}\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"AssociationType\"\u001b[0m: \u001b[38;2;0;135;0m\"ContributedTo\"\u001b[0m                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m                                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[1m}\u001b[0m                                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m}\u001b[0m,                                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m{\u001b[0m                                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[38;2;0;135;0m\"EvaluateCustomModel\"\u001b[0m,                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Training\"\u001b[0m,                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Arguments\"\u001b[0m: \u001b[1m{\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"RoleArn\"\u001b[0m: \u001b[38;2;0;135;0m\"arn:aws:iam::052150106756:role/Admin\"\u001b[0m,       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"ModelPackageConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"ModelPackageGroupArn\"\u001b[0m:                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:model-package-group/te\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mst-finetuned-models-gamma\"\u001b[0m,                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"SourceModelPackageArn\"\u001b[0m:                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:model-package/test-fin\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0metuned-models-gamma/28\"\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"ServerlessJobConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"BaseModelArn\"\u001b[0m:                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:aws:hub-content/SageMakerPublicHub/\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mModel/meta-textgeneration-llama-3-2-1b-instruct/1.10.0\"\u001b[0m,         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"AcceptEula\"\u001b[0m: true,                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"JobType\"\u001b[0m: \u001b[38;2;0;135;0m\"Evaluation\"\u001b[0m,                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"EvaluationType\"\u001b[0m: \u001b[38;2;0;135;0m\"CustomScorerEvaluation\"\u001b[0m,            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"EvaluatorArn\"\u001b[0m:                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:hub-content/F3LMYANDKW\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mPZCROJVCKMJ7TOML6QMZBZRRQOVTUL45VUK7PJ4SXA/JsonDoc/eval-lambda-t\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mest/0.0.1\"\u001b[0m                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"StoppingCondition\"\u001b[0m: \u001b[1m{\u001b[0m                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"MaxRuntimeInSeconds\"\u001b[0m: \u001b[1;36m86400\u001b[0m                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"HyperParameters\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"task\"\u001b[0m: \u001b[38;2;0;135;0m\"gen_qa\"\u001b[0m,                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"strategy\"\u001b[0m: \u001b[38;2;0;135;0m\"gen_qa\"\u001b[0m,                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"evaluation_metric\"\u001b[0m: \u001b[38;2;0;135;0m\"all\"\u001b[0m,                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"max_new_tokens\"\u001b[0m: \u001b[38;2;0;135;0m\"8192\"\u001b[0m,                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"temperature\"\u001b[0m: \u001b[38;2;0;135;0m\"0\"\u001b[0m,                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"top_k\"\u001b[0m: \u001b[38;2;0;135;0m\"-1\"\u001b[0m,                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"top_p\"\u001b[0m: \u001b[38;2;0;135;0m\"1.0\"\u001b[0m,                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"max_model_len\"\u001b[0m: \u001b[38;2;0;135;0m\"12000\"\u001b[0m,                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"aggregation\"\u001b[0m: \u001b[38;2;0;135;0m\"mean\"\u001b[0m,                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"postprocessing\"\u001b[0m: \u001b[38;2;0;135;0m\"True\"\u001b[0m                               \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"OutputDataConfig\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"S3OutputPath\"\u001b[0m:                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"s3://mufi-test-serverless-smtj/eval/\"\u001b[0m,                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[38;2;0;135;0m\"CompressionType\"\u001b[0m: \u001b[38;2;0;135;0m\"NONE\"\u001b[0m                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m}\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"InputDataConfig\"\u001b[0m: \u001b[1m[\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ChannelName\"\u001b[0m: \u001b[38;2;0;135;0m\"train\"\u001b[0m,                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"DataSource\"\u001b[0m: \u001b[1m{\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"S3DataSource\"\u001b[0m: \u001b[1m{\u001b[0m                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"S3DataType\"\u001b[0m: \u001b[38;2;0;135;0m\"S3Prefix\"\u001b[0m,                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"S3Uri\"\u001b[0m:                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"s3://sagemaker-us-west-2-052150106756/studio-users/d20251107t19\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m5443/datasets/2025-11-07T19-55-37-609Z/zc_test.jsonl\"\u001b[0m            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m                                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[1m}\u001b[0m                                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m}\u001b[0m,                                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m{\u001b[0m                                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[38;2;0;135;0m\"AssociateLineage\"\u001b[0m,                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Lineage\"\u001b[0m,                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"DependsOn\"\u001b[0m: \u001b[1m[\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"CreateEvaluationAction\"\u001b[0m                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[1m]\u001b[0m,                                                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[38;2;0;135;0m\"Arguments\"\u001b[0m: \u001b[1m{\u001b[0m                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"Artifacts\"\u001b[0m: \u001b[1m[\u001b[0m                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ArtifactName\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"-\"\u001b[0m,                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m{\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m}\u001b[0m,                                             \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"custom-eval-report\"\u001b[0m                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m]\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"ArtifactType\"\u001b[0m: \u001b[38;2;0;135;0m\"EvaluationReport\"\u001b[0m,                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"SourceUri\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Get\"\u001b[0m:                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0m\"Steps.EvaluateCustomModel.OutputDataConfig.S3OutputPath\"\u001b[0m        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m                                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m,                                                       \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[38;2;0;135;0m\"Associations\"\u001b[0m: \u001b[1m[\u001b[0m                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m{\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Source\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Name\"\u001b[0m: \u001b[1m{\u001b[0m                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"-\"\u001b[0m,                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m{\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                               \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m}\u001b[0m,                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"custom-eval-report\"\u001b[0m                         \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m]\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m}\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m,                                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Type\"\u001b[0m: \u001b[38;2;0;135;0m\"Artifact\"\u001b[0m                                 \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"Destination\"\u001b[0m: \u001b[1m{\u001b[0m                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[38;2;0;135;0m\"Arn\"\u001b[0m: \u001b[1m{\u001b[0m                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[38;2;0;135;0m\"Std:Join\"\u001b[0m: \u001b[1m{\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"On\"\u001b[0m: \u001b[38;2;0;135;0m\"/\"\u001b[0m,                                     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[38;2;0;135;0m\"Values\"\u001b[0m: \u001b[1m[\u001b[0m                                    \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[38;2;0;135;0m\"arn:aws:sagemaker:us-west-2:052150106756:ac\u001b[0m \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;0;135;0mtion\"\u001b[0m,                                                           \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m{\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                               \u001b[38;2;0;135;0m\"Get\"\u001b[0m: \u001b[38;2;0;135;0m\"Execution.PipelineExecutionId\"\u001b[0m     \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                             \u001b[1m}\u001b[0m                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                           \u001b[1m]\u001b[0m                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                         \u001b[1m}\u001b[0m                                                \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                       \u001b[1m}\u001b[0m                                                  \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[1m}\u001b[0m,                                                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                     \u001b[38;2;0;135;0m\"AssociationType\"\u001b[0m: \u001b[38;2;0;135;0m\"ContributedTo\"\u001b[0m                   \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                   \u001b[1m}\u001b[0m                                                      \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m                 \u001b[1m]\u001b[0m                                                        \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m               \u001b[1m}\u001b[0m                                                          \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m             \u001b[1m}\u001b[0m                                                            \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m           \u001b[1m]\u001b[0m                                                              \u001b[2m                     \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[1m}\u001b[0m                                                                \u001b[2m                     \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> No existing pipeline found with prefix                                <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#212\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">212</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         SagemakerEvaluation-CustomScorerEvaluation, creating new one          <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m No existing pipeline found with prefix                                \u001b]8;id=437465;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=501901;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#212\u001b\\\u001b[2m212\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         SagemakerEvaluation-CustomScorerEvaluation, creating new one          \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Creating new pipeline:                                                 <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#57\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">57</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         SagemakerEvaluation-CustomScorerEvaluation-<span style=\"color: #ffff00; text-decoration-color: #ffff00\">1c2e4a67-ecb4-4c89-8e82-e82</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">               </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #ffff00; text-decoration-color: #ffff00\">3cbe579c3</span>                                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">               </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Creating new pipeline:                                                 \u001b]8;id=91501;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=923226;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#57\u001b\\\u001b[2m57\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         SagemakerEvaluation-CustomScorerEvaluation-\u001b[93m1c2e4a67-ecb4-4c89-8e82-e82\u001b[0m \u001b[2m               \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[93m3cbe579c3\u001b[0m                                                              \u001b[2m               \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Creating pipeline resource.                                         <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/resources.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">resources.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/resources.py#30147\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">30147</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Creating pipeline resource.                                         \u001b]8;id=877192;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/resources.py\u001b\\\u001b[2mresources.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=410393;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/resources.py#30147\u001b\\\u001b[2m30147\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Successfully created pipeline:                                         <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#76\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">76</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         SagemakerEvaluation-CustomScorerEvaluation-<span style=\"color: #ffff00; text-decoration-color: #ffff00\">1c2e4a67-ecb4-4c89-8e82-e82</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">               </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #ffff00; text-decoration-color: #ffff00\">3cbe579c3</span>                                                              <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">               </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Successfully created pipeline:                                         \u001b]8;id=802515;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=256656;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#76\u001b\\\u001b[2m76\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         SagemakerEvaluation-CustomScorerEvaluation-\u001b[93m1c2e4a67-ecb4-4c89-8e82-e82\u001b[0m \u001b[2m               \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[93m3cbe579c3\u001b[0m                                                              \u001b[2m               \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Waiting for pipeline                                                   <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#79\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">79</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         SagemakerEvaluation-CustomScorerEvaluation-<span style=\"color: #ffff00; text-decoration-color: #ffff00\">1c2e4a67-ecb4-4c89-8e82-e82</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">               </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #ffff00; text-decoration-color: #ffff00\">3cbe579c3</span> to be ready<span style=\"color: #808000; text-decoration-color: #808000\">...</span>                                               <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">               </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Waiting for pipeline                                                   \u001b]8;id=984002;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=40351;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#79\u001b\\\u001b[2m79\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         SagemakerEvaluation-CustomScorerEvaluation-\u001b[93m1c2e4a67-ecb4-4c89-8e82-e82\u001b[0m \u001b[2m               \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[93m3cbe579c3\u001b[0m to be ready\u001b[33m...\u001b[0m                                               \u001b[2m               \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">/Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/rich/live.py:231: UserWarning: \n",
       "install \"ipywidgets\" for Jupyter support\n",
       "  warnings.warn('install \"ipywidgets\" for Jupyter support')\n",
       "</pre>\n"
      ],
      "text/plain": [
       "/Users/mufi/.local/share/mise/installs/python/3.12.12/lib/python3.12/site-packages/rich/live.py:231: UserWarning: \n",
       "install \"ipywidgets\" for Jupyter support\n",
       "  warnings.warn('install \"ipywidgets\" for Jupyter support')\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Final Resource Status: <span style=\"font-weight: bold\">Active</span>                                       <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/resources.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">resources.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/resources.py#30410\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">30410</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Final Resource Status: \u001b[1mActive\u001b[0m                                       \u001b]8;id=750224;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/resources.py\u001b\\\u001b[2mresources.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=46929;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-core/src/sagemaker/core/resources.py#30410\u001b\\\u001b[2m30410\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Pipeline                                                               <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#82\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">82</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         SagemakerEvaluation-CustomScorerEvaluation-<span style=\"color: #ffff00; text-decoration-color: #ffff00\">1c2e4a67-ecb4-4c89-8e82-e82</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">               </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #ffff00; text-decoration-color: #ffff00\">3cbe579c3</span> is now active and ready for execution                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">               </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Pipeline                                                               \u001b]8;id=674167;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=265281;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#82\u001b\\\u001b[2m82\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         SagemakerEvaluation-CustomScorerEvaluation-\u001b[93m1c2e4a67-ecb4-4c89-8e82-e82\u001b[0m \u001b[2m               \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[93m3cbe579c3\u001b[0m is now active and ready for execution                        \u001b[2m               \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Starting pipeline execution: eval-meta-1b49b716-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1764452564</span>            <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#263\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">263</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Starting pipeline execution: eval-meta-1b49b716-\u001b[1;36m1764452564\u001b[0m            \u001b]8;id=27465;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=541837;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#263\u001b\\\u001b[2m263\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 13:42:45] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Pipeline execution started:                                           <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#274\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">274</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         -CustomScorerEvaluation-<span style=\"color: #ffff00; text-decoration-color: #ffff00\">1c2e4a67-ecb4-4c89-8e82-e823cbe579c3</span>/executio <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         n/u2q2dl1w5aiq                                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 13:42:45]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Pipeline execution started:                                           \u001b]8;id=368377;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=144012;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#274\u001b\\\u001b[2m274\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         -CustomScorerEvaluation-\u001b[93m1c2e4a67-ecb4-4c89-8e82-e823cbe579c3\u001b[0m/executio \u001b[2m                \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         n/u2q2dl1w5aiq                                                        \u001b[2m                \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "✓ Evaluation execution started successfully!\n",
      "  Execution Name: eval-meta-1b49b716\n",
      "  Pipeline Execution ARN: arn:aws:sagemaker:us-west-2:052150106756:pipeline/SagemakerEvaluation-CustomScorerEvaluation-1c2e4a67-ecb4-4c89-8e82-e823cbe579c3/execution/u2q2dl1w5aiq\n",
      "  Status: Executing\n"
     ]
    }
   ],
   "source": [
    "# Start evaluation\n",
    "execution = evaluator.evaluate()\n",
    "\n",
    "print(\"\\n✓ Evaluation execution started successfully!\")\n",
    "print(f\"  Execution Name: {execution.name}\")\n",
    "print(f\"  Pipeline Execution ARN: {execution.arn}\")\n",
    "print(f\"  Status: {execution.status.overall_status}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monitor Job Progress\n",
    "\n",
    "Use `refresh()` to update the job status, or `wait()` to block until completion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current Status: Executing\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">PipelineExecutionStatus</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">overall_status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Executing'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">step_details</span>=<span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">StepDetail</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">name</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'EvaluateCustomModel'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Executing'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">start_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-29T13:42:45.523000-08:00'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">end_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'</span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">&lt;sagemaker.core.utils.utils.Unassigned</span><span style=\"color: #008700; text-decoration-color: #008700\"> object at 0x120ab8f80</span><span style=\"color: #008700; text-decoration-color: #008700; font-weight: bold\">&gt;</span><span style=\"color: #008700; text-decoration-color: #008700\">'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">display_name</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #e100e1; text-decoration-color: #e100e1; font-weight: bold\">StepDetail</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">name</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'CreateEvaluationAction'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">status</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'Succeeded'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">start_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-29T13:42:45.523000-08:00'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">end_time</span>=<span style=\"color: #008700; text-decoration-color: #008700\">'2025-11-29T13:42:48.017000-08:00'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">display_name</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   │   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">]</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #d7af00; text-decoration-color: #d7af00\">failure_reason</span>=<span style=\"color: #e100e1; text-decoration-color: #e100e1; font-style: italic\">None</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;38;2;225;0;225mPipelineExecutionStatus\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0moverall_status\u001b[0m=\u001b[38;2;0;135;0m'Executing'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mstep_details\u001b[0m=\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1;38;2;225;0;225mStepDetail\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mname\u001b[0m=\u001b[38;2;0;135;0m'EvaluateCustomModel'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstatus\u001b[0m=\u001b[38;2;0;135;0m'Executing'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstart_time\u001b[0m=\u001b[38;2;0;135;0m'2025-11-29T13:42:45.523000-08:00'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mend_time\u001b[0m=\u001b[38;2;0;135;0m'\u001b[0m\u001b[1;38;2;0;135;0m<\u001b[0m\u001b[1;38;2;0;135;0msagemaker.core.utils.utils.Unassigned\u001b[0m\u001b[38;2;0;135;0m object at 0x120ab8f80\u001b[0m\u001b[1;38;2;0;135;0m>\u001b[0m\u001b[38;2;0;135;0m'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mdisplay_name\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1;38;2;225;0;225mStepDetail\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mname\u001b[0m=\u001b[38;2;0;135;0m'CreateEvaluationAction'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstatus\u001b[0m=\u001b[38;2;0;135;0m'Succeeded'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mstart_time\u001b[0m=\u001b[38;2;0;135;0m'2025-11-29T13:42:45.523000-08:00'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mend_time\u001b[0m=\u001b[38;2;0;135;0m'2025-11-29T13:42:48.017000-08:00'\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mdisplay_name\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m,\n",
       "\u001b[2;32m│   │   │   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m]\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[38;2;215;175;0mfailure_reason\u001b[0m=\u001b[3;38;2;225;0;225mNone\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check current status\n",
    "execution.refresh()\n",
    "print(f\"Current Status: {execution.status.overall_status}\")\n",
    "\n",
    "pprint(execution.status)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wait for Completion\n",
    "\n",
    "Block execution until the job completes. This provides a rich visual experience in Jupyter notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭─────────────────────────────────────────── </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">Pipeline Execution Status</span><span style=\"color: #000080; text-decoration-color: #000080\"> ───────────────────────────────────────────╮</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Overall Status       </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\">Succeeded</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> </span>                                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Target Status        </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0; font-weight: bold\">Succeeded</span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> </span>                                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> Elapsed Time         </span><span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\"> 0.9s      </span>                                                                               <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Pipeline Steps</span>                                                                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Step Name                       Status           Duration     </span>                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\"> AssociateLineage               </span><span style=\"color: #808000; text-decoration-color: #808000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">Succeeded</span><span style=\"color: #808000; text-decoration-color: #808000\">       </span><span style=\"color: #008000; text-decoration-color: #008000\"> 1.9s         </span>                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\"> EvaluateCustomModel            </span><span style=\"color: #808000; text-decoration-color: #808000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">Succeeded</span><span style=\"color: #808000; text-decoration-color: #808000\">       </span><span style=\"color: #008000; text-decoration-color: #008000\"> 7462.5s      </span>                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span> <span style=\"color: #008080; text-decoration-color: #008080\"> CreateEvaluationAction         </span><span style=\"color: #808000; text-decoration-color: #808000\"> </span><span style=\"color: #008000; text-decoration-color: #008000\">Succeeded</span><span style=\"color: #808000; text-decoration-color: #808000\">       </span><span style=\"color: #008000; text-decoration-color: #008000\"> 2.5s         </span>                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m╭─\u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;34mPipeline Execution Status\u001b[0m\u001b[34m \u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m \u001b[0m\u001b[1;36mOverall Status      \u001b[0m\u001b[1;36m \u001b[0m\u001b[37m \u001b[0m\u001b[1;37mSucceeded\u001b[0m\u001b[37m \u001b[0m                                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m \u001b[0m\u001b[1;36mTarget Status       \u001b[0m\u001b[1;36m \u001b[0m\u001b[37m \u001b[0m\u001b[1;37mSucceeded\u001b[0m\u001b[37m \u001b[0m                                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;36m \u001b[0m\u001b[1;36mElapsed Time        \u001b[0m\u001b[1;36m \u001b[0m\u001b[37m \u001b[0m\u001b[37m0.9s     \u001b[0m\u001b[37m \u001b[0m                                                                               \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;35mPipeline Steps\u001b[0m                                                                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[1;35m \u001b[0m\u001b[1;35mStep Name                     \u001b[0m\u001b[1;35m \u001b[0m\u001b[1;35m \u001b[0m\u001b[1;35mStatus         \u001b[0m\u001b[1;35m \u001b[0m\u001b[1;35m \u001b[0m\u001b[1;35mDuration    \u001b[0m\u001b[1;35m \u001b[0m                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m \u001b[0m\u001b[36mAssociateLineage              \u001b[0m\u001b[36m \u001b[0m\u001b[33m \u001b[0m\u001b[32mSucceeded\u001b[0m\u001b[33m      \u001b[0m\u001b[33m \u001b[0m\u001b[32m \u001b[0m\u001b[32m1.9s        \u001b[0m\u001b[32m \u001b[0m                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m \u001b[0m\u001b[36mEvaluateCustomModel           \u001b[0m\u001b[36m \u001b[0m\u001b[33m \u001b[0m\u001b[32mSucceeded\u001b[0m\u001b[33m      \u001b[0m\u001b[33m \u001b[0m\u001b[32m \u001b[0m\u001b[32m7462.5s     \u001b[0m\u001b[32m \u001b[0m                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m \u001b[36m \u001b[0m\u001b[36mCreateEvaluationAction        \u001b[0m\u001b[36m \u001b[0m\u001b[33m \u001b[0m\u001b[32mSucceeded\u001b[0m\u001b[33m      \u001b[0m\u001b[33m \u001b[0m\u001b[32m \u001b[0m\u001b[32m2.5s        \u001b[0m\u001b[32m \u001b[0m                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 16:21:36] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Final Resource Status: Succeeded                                      <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">execution.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#979\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">979</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 16:21:36]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Final Resource Status: Succeeded                                      \u001b]8;id=693225;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py\u001b\\\u001b[2mexecution.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=873243;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/evaluate/execution.py#979\u001b\\\u001b[2m979\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Final Status: Succeeded\n"
     ]
    }
   ],
   "source": [
    "# Wait for job to complete (with rich visual feedback)\n",
    "execution.wait(poll=30, timeout=3600)\n",
    "\n",
    "print(f\"\\nFinal Status: {execution.status.overall_status}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 16:21:42] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> S3 bucket: mufi-test-serverless-smtj, prefix: eval           <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#130\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">130</span></a>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 16:21:42]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m S3 bucket: mufi-test-serverless-smtj, prefix: eval           \u001b]8;id=425698;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=639097;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#130\u001b\\\u001b[2m130\u001b[0m\u001b]8;;\u001b\\\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Extracted training job name:                                  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#63\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">63</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         pipelines-u2q2dl1w5aiq-EvaluateCustomModel-FNSg2Knqlf from    <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                        </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         step: EvaluateCustomModel                                     <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                        </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Extracted training job name:                                  \u001b]8;id=993672;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=652226;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#63\u001b\\\u001b[2m63\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         pipelines-u2q2dl1w5aiq-EvaluateCustomModel-FNSg2Knqlf from    \u001b[2m                        \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         step: EvaluateCustomModel                                     \u001b[2m                        \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Searching for results_*.json in                              <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#150\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">150</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         s3:<span style=\"color: #e100e1; text-decoration-color: #e100e1\">//mufi-test-serverless-smtj/eval/pipelines-u2q2dl1w5aiq-E</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #e100e1; text-decoration-color: #e100e1\">valuateCustomModel-FNSg2Knqlf/output/output/</span>                 <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Searching for results_*.json in                              \u001b]8;id=724854;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=324888;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#150\u001b\\\u001b[2m150\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         s3:\u001b[38;2;225;0;225m/\u001b[0m\u001b[38;2;225;0;225m/mufi-test-serverless-smtj/eval/pipelines-u2q2dl1w5aiq-E\u001b[0m \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[38;2;225;0;225mvaluateCustomModel-FNSg2Knqlf/output/output/\u001b[0m                 \u001b[2m                         \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Found results file:                                          <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#168\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">168</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         eval/pipelines-u2q2dl1w5aiq-EvaluateCustomModel-FNSg2Knqlf/o <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         utput/output/eval-meta_textgeneration_llama_3_2_1b_instruct- <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         -or8pa/eval_results/results_2025-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">11</span>-29T23-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">46</span>-<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">45.108093</span>+<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">00-00</span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">.j</span>son                                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m                   \u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Found results file:                                          \u001b]8;id=770358;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=338226;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#168\u001b\\\u001b[2m168\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         eval/pipelines-u2q2dl1w5aiq-EvaluateCustomModel-FNSg2Knqlf/o \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         utput/output/eval-meta_textgeneration_llama_3_2_1b_instruct- \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         -or8pa/eval_results/results_2025-\u001b[1;36m11\u001b[0m-29T23-\u001b[1;36m46\u001b[0m-\u001b[1;36m45.108093\u001b[0m+\u001b[1;36m00-00\u001b[0m \u001b[2m                         \u001b[0m\n",
       "\u001b[2;36m                    \u001b[0m         \u001b[1;36m.j\u001b[0mson                                                        \u001b[2m                         \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">[11/29/25 16:21:43] </span><span style=\"color: #0069ff; text-decoration-color: #0069ff; font-weight: bold\">INFO    </span> Using metrics from key: <span style=\"color: #008700; text-decoration-color: #008700\">'custom|gen_qa_gen_qa|0'</span> <span style=\"font-weight: bold\">(</span>gen_qa or  <a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">show_results_utils.py</span></a><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">:</span><a href=\"file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#100\" target=\"_blank\"><span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">100</span></a>\n",
       "<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">                    </span>         custom_scorer format<span style=\"font-weight: bold\">)</span>                                        <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                         </span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[2;36m[11/29/25 16:21:43]\u001b[0m\u001b[2;36m \u001b[0m\u001b[1;38;2;0;105;255mINFO    \u001b[0m Using metrics from key: \u001b[38;2;0;135;0m'custom|gen_qa_gen_qa|0'\u001b[0m \u001b[1m(\u001b[0mgen_qa or  \u001b]8;id=904034;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py\u001b\\\u001b[2mshow_results_utils.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=137242;file:///Volumes/workplace/sagemaker-python-sdk-staging/sagemaker-train/src/sagemaker/train/common_utils/show_results_utils.py#100\u001b\\\u001b[2m100\u001b[0m\u001b]8;;\u001b\\\n",
       "\u001b[2;36m                    \u001b[0m         custom_scorer format\u001b[1m)\u001b[0m                                        \u001b[2m                         \u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-style: italic\">                </span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold; font-style: italic\">Custom Model Results</span><span style=\"font-style: italic\">                </span>\n",
       "╭────────────────────────────────┬─────────────────╮\n",
       "│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> Metric                         </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">           Value </span>│\n",
       "├────────────────────────────────┼─────────────────┤\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> bleu                           </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          6.6928 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> bleu_stderr                    </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.7769 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> byoc_failure_count             </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">       3572.0000 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> em                             </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">           1.26% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> em_stderr                      </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0019 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> f1                             </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          19.13% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> f1_score_quasi                 </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          25.29% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> f1_score_quasi_stderr          </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0049 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> f1_stderr                      </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0047 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> qem                            </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">           2.21% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> qem_stderr                     </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0025 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rouge1                         </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          25.73% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rouge1_stderr                  </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0047 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rouge2                         </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          19.15% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rouge2_stderr                  </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0047 </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rougeL                         </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          25.04% </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\"> rougeL_stderr                  </span>│<span style=\"color: #c0c0c0; text-decoration-color: #c0c0c0\">          0.0047 </span>│\n",
       "╰────────────────────────────────┴─────────────────╯\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[3m                \u001b[0m\u001b[1;3;32mCustom Model Results\u001b[0m\u001b[3m                \u001b[0m\n",
       "╭────────────────────────────────┬─────────────────╮\n",
       "│\u001b[1;32m \u001b[0m\u001b[1;32mMetric                        \u001b[0m\u001b[1;32m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m          Value\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "├────────────────────────────────┼─────────────────┤\n",
       "│\u001b[36m \u001b[0m\u001b[36mbleu                          \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         6.6928\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mbleu_stderr                   \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.7769\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mbyoc_failure_count            \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m      3572.0000\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mem                            \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m          1.26%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mem_stderr                     \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0019\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mf1                            \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         19.13%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mf1_score_quasi                \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         25.29%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mf1_score_quasi_stderr         \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0049\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mf1_stderr                     \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0047\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mqem                           \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m          2.21%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mqem_stderr                    \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0025\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrouge1                        \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         25.73%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrouge1_stderr                 \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0047\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrouge2                        \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         19.15%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrouge2_stderr                 \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0047\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrougeL                        \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         25.04%\u001b[0m\u001b[37m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36mrougeL_stderr                 \u001b[0m\u001b[36m \u001b[0m│\u001b[37m \u001b[0m\u001b[37m         0.0047\u001b[0m\u001b[37m \u001b[0m│\n",
       "╰────────────────────────────────┴─────────────────╯\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080\">╭─────────────────────────────────────────── </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">Result Artifacts Location</span><span style=\"color: #000080; text-decoration-color: #000080\"> ───────────────────────────────────────────╮</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>  <span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">📦 </span><span style=\"font-weight: bold\">Full evaluation artifacts available at:</span>                                                                     <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>  <span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">Custom Model:</span>                                                                                                  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>  <span style=\"color: #008080; text-decoration-color: #008080\">  s3://mufi-test-serverless-smtj/eval/pipelines-u2q2dl1w5aiq-EvaluateCustomModel-FNSg2Knqlf/output/output/Non</span>  <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>  <span style=\"color: #008080; text-decoration-color: #008080\">e/eval_results/</span>                                                                                                <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">│</span>                                                                                                                 <span style=\"color: #000080; text-decoration-color: #000080\">│</span>\n",
       "<span style=\"color: #000080; text-decoration-color: #000080\">╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[34m╭─\u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m \u001b[0m\u001b[1;34mResult Artifacts Location\u001b[0m\u001b[34m \u001b[0m\u001b[34m──────────────────────────────────────────\u001b[0m\u001b[34m─╮\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m  \u001b[1;34m📦 \u001b[0m\u001b[1mFull evaluation artifacts available at:\u001b[0m                                                                     \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m  \u001b[1;32mCustom Model:\u001b[0m                                                                                                  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m  \u001b[36m  s3://mufi-test-serverless-smtj/eval/pipelines-u2q2dl1w5aiq-EvaluateCustomModel-FNSg2Knqlf/output/output/Non\u001b[0m  \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m  \u001b[36me/eval_results/\u001b[0m                                                                                                \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m│\u001b[0m                                                                                                                 \u001b[34m│\u001b[0m\n",
       "\u001b[34m╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# show results\n",
    "execution.show_results()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieve Existing Job\n",
    "\n",
    "You can retrieve a previously started evaluation job using its ARN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO - sagemaker.modules.evaluate.execution - Extracted s3_output_path from training job pipelines-amlk8q2ukw8x-EvaluateCustomModel-VElzvyVY19: s3://mufi-test-serverless-smtj/eval/\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieved job: amlk8q2ukw8x\n",
      "Status: Succeeded\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.train.evaluate import EvaluationPipelineExecution\n",
    "\n",
    "# Get existing job by ARN\n",
    "existing_arn = execution.arn  # Or use a specific ARN\n",
    "\n",
    "existing_exec = EvaluationPipelineExecution.get(arn=existing_arn)\n",
    "\n",
    "print(f\"Retrieved job: {existing_exec.name}\")\n",
    "print(f\"Status: {existing_exec.status.overall_status}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List All Custom Scorer Evaluations\n",
    "\n",
    "Retrieve all custom scorer evaluation executions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 0 custom scorer evaluation(s):\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Get all custom scorer evaluations\n",
    "all_executions = list(CustomScorerEvaluator.get_all())\n",
    "\n",
    "print(f\"Found {len(all_executions)} custom scorer evaluation(s):\\n\")\n",
    "for execution in all_executions:\n",
    "    print(f\"  - {execution.name} - {execution.arn}: {execution.status.overall_status}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stop a Running Job (Optional)\n",
    "\n",
    "You can stop a running evaluation if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment to stop the job\n",
    "# execution.stop()\n",
    "# print(f\"Execution stopped. Status: {execution.status.overall_status}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "This notebook demonstrated:\n",
    "1. ✅ Creating a CustomScorerEvaluator with a custom evaluator ARN\n",
    "2. ✅ Starting an evaluation job\n",
    "3. ✅ Monitoring job progress with refresh() and wait()\n",
    "4. ✅ Retrieving existing jobs\n",
    "5. ✅ Listing all custom scorer evaluations\n",
    "\n",
    "### Key Points:\n",
    "- The `evaluator` parameter accepts:\n",
    "  - Custom evaluator ARN (for AI Registry evaluators)\n",
    "  - Built-in metric names (\"code_executions\", \"math_answers\", \"exact_match\")\n",
    "  - Evaluator objects from sagemaker.ai_registry.evaluator.Evaluator\n",
    "- Set `evaluate_base_model=False` to only evaluate the custom model\n",
    "- Use `execution.wait()` for automatic monitoring with rich visual feedback\n",
    "- Use `execution.refresh()` for manual status updates\n",
    "- The SageMaker session is automatically inferred from your environment"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
